The Mdl Principle, Penalized Likelihoods, and Statistical Risk

We determine, for both countable and uncountable collections of functions, informationtheoretic conditions on a penalty pen(f) such that the optimizer f̂ of the penalized log likelihood criterion log 1/likelihood(f) + pen(f) has statistical risk not more than the index of resolvability corresponding to the accuracy of the optimizer of the expected value of the criterion. If F is the linear span of a dictionary of functions, traditional description-length penalties are based on the number of non-zero terms of candidate fits (the `0 norm of the coefficients) as we review. We specialize our general conclusions to show the `1 norm of the coefficients times a suitable multiplier λ is also an information-theoretically valid penalty.